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INTRODUCTION 



Mr. Chairman, my name is Kenneth Komoski. I am 
President of the Educational Products Information 
Exchnage Institute. The Institute, usually 
referred to as EPIE (pronounced "eppy"), was 
chartered in 1967 by the Regents of the State 
University of New York as a nonprofit corporation. 

It is a consumers’ union for member schools and 
school systems in 50 states. With support from 
these schools, other members, and foundation 
grants, the Institute conducts independent studies 
of all types of educational materials and equipment. 
It publishes it findings in nine Educationa l 
Product Reports each year. At present, these 
reports reach about 3,500 educators in schools, 

colleges, and other institutions. 

'V-. 



Prior to organizing EPIE, I served as co-director 
of the Institute for Educational Technology at 
Columbia University. While in that position, I 
edited the first publication to document the fact 

that many educational materials which schools assumed 
had been developed through a process of field 
testing and revision - a process which EPIE has 
labeled "learner verification" - had, in fact, 
never been tried out and revised on the basis of 
feedback from learners prior to publication. 

Before joining the University, I directed a non- 
profit research and development organization 
which had pioneered in the creation of learn- 
verified educational materials. I have also 
been a classroom teacher, the head of a junior 

high school, and a special consultant on educational 
technology to UNESCO and to the U.S. State Department. 

During the past year, I have served as an advisor 
to the Rand Corporation *s Study for the National 
Institute of Education and as the chairman of the 
forum on Educational Technology at the 1970 
White House Conference on Children. 



My research efforts in educational technology 
have focused primarily on improving student learning 
by improving the effectiveness of the materials 
from which students are expected to learn. 



I believe that 



POINT 1.] 



Educational technology: 
effectiveness then 
efficiency 



unless educational technology is focused first 
and foremost on the improvement of learning, 
it becomes simply a means of making education 
seem more efficient without being truly effective. 
Once the difficult goal of effectiveness is being 
consistently achieved, education can easily be made 
more efficient. To proceed in any other fashion 
means leaving the ultimate educational consumer - 
the learner - in a constantly vulnerable position. 



One important function of the proposed National 
Institute of Education must be to conduct research 
and provide leadership that will guarantee all 
American citizens that every educational material 
from which children are required to learn is as 
effective as we know how to make it . 



What I have to say to you this morning will 
clearly indicate that there is an enormous need 
for better educational materials, and an even 
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greater need for procedures that will guarantee 
their continuous improvement. The proposed 
National Institute could pi-ovide many of the means 
and methods upon which that improvement depends . 

TOE PRESENT SITU ATION - WHAT SCHOOLS USE AN D WHY 

POINT 2.1 

50 Million children 
are unprotected from 
inadequately developed 
educational materials 



My review of the present situation regarding 

e evelopment) selection and, use of educational 
materials has led me to the following conclusion: 
t this time of national concern over con- 
sumer protection, the largest single grouD of 
unprotected consumers is made up of tl^e 
million school children who are being required 
to learn from educational materials almost 
all of which have been inadequately developed 
and evaluated. 



The materials I refer to are the textbooks , 
films, tapes, kits, and complex multimedia 
systems that are the concrete artifacts of 
learning in our schools . These materials are 
the tools of learning, constantly used and 
depended upon by teachers and students alike. 
Indeed, one can conceive of a school without 
walls, or even an educational system without 
schools, but a modern educational system without 
tools — in the form of a variety of educational 
materials— is inconceivable. Today’s education 
requires that these tools be better, sharper, 
more dependable— in other words, more effective 
than ever before. 4 : 



POINT 2.2 

Improved materials 
are urgently needed 
because of growing 
demands for: 
individualized 
learning; greater 
teacher account- 
ability 



This requirement is urgent. The interaction of 
two growing trends in education demand it. The 
first is the trend toward more independent, 
individualized learning on the part of students . ^ 
The second is the demand for greater accountability 
on the part of teachers. As these two trends 
continue to build, the need for materials of 
demonstrated dependability and effectiveness will 
become more and more acute. If materials with 
these qualities are not generally available, 
neither students nor teachers can be expected 
to meet the new demands being made on them. 



We are only now beginning to comprehend the need 
for learner-oriented, individualized educational 
materials. For decades educators have talked about 
the fact that learning is an individual act and that 
each learner must ultimately have an individualized 
program. Today there is growing evidence that this 
talk is finally being translated into action. But 
those educators who are acting - by establishing 
new types of individualized learning environments - 
are beginning to realize that the quality of the 
materials they put into the hands of learners is 
a critical factor in the success of their new 
programs. When, in addition, these educators 
are likely to be held more rigorously accountable 
for the learning of each individual student, they 
are going to pay even moie attention to the quality 
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and appro prialrGness of the materials being made 

available to them. However, if higher , quality materials 
are not generally available neither teachers nor their 

students can be expected to be held completely 
accountable for learning failures. 



POINT 2,3 



Schools use in- 
adequatelniaterial s 
because they must 
use something 



But the overwhelming majority of the materials 
now used by schools are not of " the highest 
possible quality ." Thus, to answer the questions 
the Committee has put to me as succinctly as 
possible : 

"What do schools use?" — They use what’s available. 

"Why?" --Because it's available, and you can't 
run a school without materials. 



During the last two decades, nev; technologies, new 
media, and Federal money made possible enormous 
and continuing increases both in the production 
and purchasing of these traditional group-oriented 
educational materials . This is not to say that 
Federal money hasn’t also helped to develop a small 
number of new, more effective, learning materials. 

It has . But for every Federal dollar spent since 
1958 on the development of new, more effective 
materials, hundreds more have also been made avail- 
able to schools for the purchase of materials that are 
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much less effective than they should be. 



POINT 3.1 



Modern technology 
is producing an 
increasing quantity 
of trivial options - 
but teachers and 
students need high 
quality alternatives 
to these inadequate 
materials 



The result is that educational purchasers 
are being presented with a tremendous variety 
of options — even more than are offered to the 
American car buyer. But, as is the case with 
so many of those automotive options, the educator’s 
options are too often trivial variations on over- 
worked and, in the long lun, ineffective themes. 
Today’s teachers and students do not need an ever- 
increasing quantity of options. What they do need 
desperately are high quality alternatives to the 
inadequately developed materials they are now 
required to use. 



Before I turn to this crucial question of quality, 

I want to take a few minutes to examine in some 
detail the quantitive growth of educational matei?ials 
during the last two decades . My examination begins 
at a point seven years before NDEA began to prime 
the materials-purchasing pump with Federal dollars. 

In 1951, the American Educational Catalog — the only 
source that attempted a comprehensive catalog of 
materials developed for use in elementary and secondary 
schools — listed some 5,600 items* Although some 
workbooks and films were included, almost all the items 
were standard textbooks with standard formats, designed 



for use in the standard (30 student) American 
classroom by the "average" teacher teaching 
"average" students. Other less comprehensive 
catalogs may have contained additional listings 
of perhaps five to eight thousand items, mostly 
films, many of which had not been produced primarily 
for use in schools . But a safe estimate of strictly 
school-focused materials would be in the neighborhood 
of 10,000 items. 



Twenty years later, the 1971 El -Hi Textbooks 
in Print , the successor to the American Educational 
Catalog , lists about 14,000 textbooks. The 
publisher, R.R. Bowker, estimates that the catalog 
also refers to 56,000 workbooks and other 
supplementary materials marketed along with those 
textbooks. The increase appears to be ten-fold 



Bowker has also published separate catalogs, 
compiled at the University of Southe*''n California, 
of films, filmstrips, and transparencies. 
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ihese catalogs list some 55,000 additional 
items, only a few of which are included as 
nonbook supplementary materials in Textbooks 
in Print. Taken together, the mateials listed 
in all Bowker Catalogs total 126,000. 

But that IS not all, During the last year, 
Westinghouse Learning Corporation has published 
a seven-volume catalogue called Learning Directory , 
which covers all media, and lists 207,000 items. 

And even this ambitious effort inevitably falls 
short of presenting a full picture of the materials 
available to schools. Every such "comprehensive" 
catalog is obsolete the moment it is printed: 
its potential content is constantly growing both 
in number of items and in kinds of media. 



^ristance, the information you are now looking 
at is being projected from a film transparency via 
an overhead projector. In 1951, there were no 
overhead transparencies for use in schools. Today, 
a USC-Bowker catalog lists approximately 13,000 
sets of transparencies. The Westinghouse Directory 
lists 15,000 sets and EPIE estimates that there are 
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actually over 20,000 sets commercially available 
for school use. EPIE’s estimate exceeds both the 
Westinghouse and USC— Bowker listing largely because 
neither managed to identify all commercial producers 
of transparencies; indeed, EPIE is not certain that 
it has identified all of them. Furthermore, this 
estimate of 20,000 does not include the hundreds 
of thousands of transparencies that are produced 
by the schools themselves . And transparencies 
are just one of the many kinds of media which 
have been introduced into the schools since 19 51. 



In 19 51, 16 mm film had just begun to be used 
widely in schools. Other uses of film, Super 8 
film loops, 35 mm sound filmstrips, and EVR 
(Electronic Video Recording, actually film) had 
not yet appeared. Likewise, school use of audio 
tapes was some years off , and video tapes had not 
even been invented. The use of programmed instruction 
books and teaching machines was a decade away; and 
computer-assisted instruction (CAI) a bit further. 
Simulation games were even further in the future; 
and today’s most "in” materials — multimedia instrc- 
tional systems — were unheard of. 
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Today, twenty years later, schools have available 
to them at least 50,000 instructional film 
materials-- 16 mm films, 8 mm films and loops, 
and 35 mm filmstrips. 

There are upwards of 20,000 audio tapes which 
schools may purchase or rent. Thousands more are 
being produced and used locally by schools themselves. 



Although a recent survey by National Instructional 
Television indicates that approximately 85 percent 
of the instructional television being broadcast 
to schools is provided by only 223 video tapes, an 
estimated additional 10,000 one-inch, 1/2 inch 
and 1/4 inch video tapes, produced both commercially 
and locally, account for hundreds of thousands of 
nonbroadcast hours of television instruction. 




In the ten years since they were introduced pro- 
grammed instruction materials for school use have 
increased from a few dozen to approximately 4,000, 
approximately five hundred of which must be used 
in specific types of teaching machines. All of these 
programs could be adapted to CAI, and many have been. 
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simulation games for education are still in 
their infancy. However, there are now 404 
catalogued for school use, and the publisher 
of the catalog- estimates that twice that 
number are nearing completion. 



Multimedia kits are perhaps the most difficult 
instructional materials to quantify with any 
reliability. Many of them are sold as parts 
of instructional programs based on textbook series; 
others are sold to stand alone. Some are made up 
simply of a book and audio tapes , while others 
contain a great variety of assorted materials and 
equipment. Thus, it is impossible to arrive at an 
exact count, but they clearly number in the thousands. 



POINT 3.2 



There are at 
present well 
over 200,000 
education materials 
being marketed to 
schools 



Overall, the growth in all types of materials since 
1951 may be described conservatively as an increase 
of about twenty-fold; from something over 10,00 0 in 
1951 to well over 200,000 in 1971. 



I 



As I have mentioned , much of this growth was made 
possible by advances in production technology, the 
appearance of new media — and increased purchasing 
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power with Federal funds. But there have been 
other factors, as w611, ones which have contributed 
to the quantity of materials but which primarily 
are related to the question of quality. They have 
brought about materials of greater social relevence 
and materials with new teaching approaches and/or 
more valid and up-to-date content. Unfortunately, 
the factors have only indirectly contributed to 
the improvement of the learning effectiveness of 
the materials they have introduced. 



The first of these additional factors, social 
relevancy, has been responsible for the introduction 
of new subjects into the curriculum. Thus we now 
have materials in "Black Studies", in "Sex Education", 
in "Environmental Education" or "Ecology". And who 
in 1951 would have predicted that today third-grade 
pupils in elegant Public School #6 in New York City 
would be using instructional materials on the dangers 
of narcotics? 
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The second of "these factors ha.s been the appearance 
of new approaches to teaching and learning e.g., 
learning by discovery, by simulation, or, as is the 
case with the Initial Teaching Alphabet, a new 
orthography. All of these approaches have been 
quickly suppor"ted by new instructional materials, 
including many which have completely restructured 
traditional subject matter, so that we now have 
the "new math", "Project English", and the "new 
social studies", to mention a few. 



POINT 3.3 

New methods of 
developing ed- 
ucational materials 



An important handful of these new departures 

funded by Federal or private foundation 
support -- have also managed ro help pioneer 
new methods of developing educational materials. 




I 

"Biology", for example, has been a high school 
course since the nineteenth century. In 1951, it 
had been essentially unchanged since that time. 
Basically, it was a course in taxonomy, and the 
most widely used standard textbook available was 
Holt’s Modern Biology , essentially a compendium of 
taxonomic information, designed for use in the 
traditional teacher-dependent, "average" class of 
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of 30 "average" students. A decade ago, the 
Federally-supported Biological Sciences Curriculum 
Study Group began work on a new biology course. 

But because the group of prominent educators and 
biologists assembled did not agree that all students 
could best learn biology from a single approach with 
a single textbook, their work produced three separate 
approaches to the learning of biology: a "cellular" 
approach, a "biochemical" approach and an 
"ecological" approach. 



POINT 3.4 



Learner-verification 
and revision a major 
new departure in 
product development 



In addition to these new departures, the 
developers also took a new approach to the 
development of their materials: they thoroughly 

evaluated the effectiveness of their materials on 
a group of learners and revised them accordingly, 
before ^they were released for commercial distribution. 
THTs process of learner verificati on and revision 
emerged about a decade ago, primarily in the develop- 
ment of programmed instruction materials . Since 
that time the process has been adopted by many 
Federally-funded product development programs, but 
by very few commericial producers. 




One reason why commercial producers have not adopted 
the verification process may be that few schools look 
for evidence of learner- verification when selecting 
curriculum materials. This would explain, at least 
in part, why — ten years later —the non-verified 
Modern Biology still holds on to a significant portion 
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of the school market. Another reason is that 
new materials with new approaches are harder to 
sell. They often (as is the case with the BSCS 
materials) require a certain amount of teacher 
retraining} while the traditional text is as easy 
for teachers to slip into as an old shoe. 

Ab Edward Katzenbach, former President of Raytheon 
Education has put it, "The money is in the old 
stuff, not the new stuff." 



POINT 4.1 



Small size of the 
industry and small 
size of school 
expenditures 
for materials 



PRESENT STATUS OF THE EDUCATION INDUSTRY 

Last year?s total sales for all educational companies 

marketing book and nonbook media (but not including 
equipment) were no more than $60 0 million, whereas, 

185 single companies in Fortune Magazine’s list of 
the country’s top 500 had sales exceeding $600 million. 
This $800 million represents less than 2 percent of the 
country’s school budget. Or, put another way, local 
schools are spending less than 2^ of every budget 
dollar on the tools of learning. Interestingly, 
during recent years the average school system has 
received just about 2% of its budget through Federal 
support - A major reason why the fortunes of the 
educational industry have been so closely tied to the 
availability of Federal money for local schools. 



The fact of the matter is;y however, that when 
compared to other important American industries , 
relatively speaking the "real money" is just 
not in education, at least not at present. 

Today, the educational materials industry is 

small when compared to most others . 

16 
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Nevertheless , futurists and economists predict that 
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the educational industry will grow into a major 
American industry in the years a> ^ad . Assuming 
they are correct, now is the time, while 

the industry is still in its economic infancy- 
and schools are still spending relatively little 
on its products— to encourage it, cajole it, give 
it guidance, and, if necessary, to manage Federal 
support in such a way that money is available to 
help develop and purchase products of proven worth. 



Everything that can be done must be done to get 
the education industry to fulfill its potential; 
not by providing the schools with endless — but 
in the end trivial -- options, but by supplying 
effective alternatives for individual learners . 

The potential impact of its products is enormous. 
These products contain materials that introduce 
(or fail to introduce) skills, concepts, facts, 
and understanding into the minds of 50 million 
young Americans for twelve crucially important years . 
No industry in the country produces products of 
greater importance or potential . 
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As the late Robert Locke, who, as Executive 
Vice-President of the McGraw-Hill Book Company, 
was one of the most forward looking leaders in 
the education industry, wrote in a recently 
published Saturday Review article titled, ’’Has 
the Education Industry Lost Its Nerve?”: 

"The most encouraging sign is 

POINT 4.2 

that educational research is 

Industry's capacity 

to apply research finally beginning to lead instead 

findings 

of follow educational experimen- 
tation. 

’’The education companies vary 
greatly in their research and 
development abilities, but in 
general it can be said that few 
of them have any capacity at all 
to do basic research in education, 
a function that can best be under- 
taken in the universities and in 
organizations devoted solely or 
largely to research. On the other 
hand. . .industry clearly has the 
capacity to apply research findings 
to experimental product development 



Mr. Locke concluded by saying 

tt 

In fact j the chief contribution 
of industry may come through its 
ability to apply the findings of 
research to the development of 
products and services for education." 



POINT 4.3 

Present lack on 
the part of 
industry 



I am in total agreement with Mr. Locke concerning 
the education industry's capacity to apply the 
results of research findings to product development, 
but my analysis of the current status of product 
development within leading education companies 
indicates that the industry is a long way from 
doing so. The research findings to which Mr. 

Locke referred clearly indicate that the learning 
effectiveness of a product can best be improved 
through the process I have referred to as 
learner-verification and revision. 



I 
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These terms are simply a researcher's way of 
saying that the learning effectiveness of a 
product will be improved if it is taken through 
a systematic cycle of tryouts with learners 
followed by revisions based on the feedback. Such 
evaluations need not always involve large groups 
of learners. Through appropriate sampling a small 
group of "target" students can give the product 
developers ample opportunity to catch errors and 
trouble spots and to revise accordingly. 



As you can see, the learner- verification and 
revision process sounds like a common-sense 
approach to development which producers might 
have discovered for themselves long ago. Indeed, 
it has been in use in the development of stand- 
ardized tests for several decades and, as Mr. 
Locke pointed out elsewhere in his article, the 
researchers who det/eloped programmed instruction 
a decade ago were the first to apply it to the 
creation of learning materials. However, 
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four years of research by the EPIE Institute 
into almost every class of materials 
from textbooks and films to multimedia kits have 
convinced us that although this research-generated 
process is becoming more generally understood by 
educational producers, their capacity to apply it 
remains, in fact, virtually unutilized. 



POINT 4.4 

99 percent of 
•present materials 
'are inadequately 
developed 



As a result we estimate that 99 percent of the 
materials school children are now required to 
use have not been put through even the initial 
phases of the learner— verification and revision 
cycle. If this statistic is disturbing, the 
picture in particular product areas is even 
more so. For instance, our investigation in 
the area of textbooks indicates that under 
one percent of the approximately 14,000 text- 
books being sold to schools has been systematically 
shaped through the learner tryout and revision 
process . 



Our study included an analysis of the best selling 
texts -plus -media-supplements in major elementary- 
high school curriculum areas. This "best seller" 
list eventually grew to some sixty different texts 
and their related materials . Fewer than 10 percent 
of these had even been field-tested prior to pub- 
lication! I say "even" because the field testing 
of textbooks is rarely synonomous with learner- 
verification and revision. In some cases, for instance. 
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reactions from salesmen in the field are referred 
to as "field testing". But when "field testing" 
refers to actual tests of the materials with students » 
it is done usually just prior to publication with no 
chance of using the results to impro«7e the product. 

Such testing is done in the hope of impressing 
purchasers that the materials in question have 
actually been used in schools . Thus it is important 
not to confuse traditional field testing of text- 
books with the learner -verification and revision process. 

Extrapolating from these sixty "best sellers" 
to all 14,000 textbooks we may estimate that ten 
percent or 1,40 0 of these have been "field tested", 
in some fashion, but the percentage that have 
been through learner-verification and revision 
is minuscule. 

In the educational film field, the amount of 
field testing and/or learner-verification and 
revision is even less . In the area of broad- 
cast video taped instruction, where EPIE has 
drawn upon the research of the National Center 
for Instructional Television, we discovered that only 
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drawn upon the research of the National Center 
for Instructional Television, we discovered that only 
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three of the 223 materials used in over 8 5 per- 
cent of broadcast instructional television have 
been learner-tested. In other words, only a little 
over one percent of the television material used 
in schools has been learner-verified. 



The director of the National Center for Audio Tapes 
at the University of Colorado told EPIE that, while 
he had no statistical data, it is his opinion that 
practically none of the estimated 20,00 0 tapes now 
available has gone through the verification-revision 
process . 




By far the most discouraging area we have investigated 
is that of programed instruction, as Mr. Locke and 
others have pointed out, research in programed in- 
struction did much to develop and refine the process 
of learnei^verification and revision. One would, 
therefore, expect to find that all (or at least the 
great majority) of such materials would have been 
thoroughly tested. Such is not the case. EPIE’s 
examination of 633 of the programed items now in use 
in major curriculum areas in schools revealed that 
research evidence "was available for only 7 per- 
cent of these materials ^ while some "field testing" 
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was claimed for another 8 percent. A cursory 
examination of the remaining 3,000 programs 
less central to the school curriculum indicates that 
even a smaller percentage of those seem to have been 
learner-verified . 

Recently, EPIE surveyv^d a sampling of major educational 
producers — some of whom produce programed materials 
and all of whom have begun to move in the direction 
of "systems of materials” involving a multitude of 
media and methods . We got in touch with them in order 
to ascertain their present attitude toward field 

testing, learner-verification and product evaluation. 

Here is a sampling of their comments as 
reported by our telephone interviewer: 



POINT 4.5 



Interviews with 
producers indicate 
lack of attention 
to testing of materials 



(Company A, Vice President and Editor-in-Chief ) 

"A couple of years ago, we wanted to do some 
field testing, but scheduling wouldn't allow 
it... It takes too much time and we wouldn't 
have gotten the books out... It wasn't a 
question^ of money, but just scheduling. We're 
now working on a program we plan to field-test. 
I hppe we can. . .Testing has lots of problems, 
you know .... 

(This company has no information on field 
testing available to schools) 
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( Company B, Editor-in-Chief ) 

"We don't do any real testing from the 
standpoint of content or pedagogy... 

When I was Editor-in-Chief at , 

we did a lot of testing. . . , but we were 
testing the format, you might say... We 
found that some difficult-to-produce 
stuff wasn't necessary... This is the 
kind of testing most publishers do; it 
can lead to some improvements from the 
teaching standpoint, but that's just 
serendipity . " 

"More testing is needed, but it costs a lot.. 
When I was teaching, I always wanted to know 
about classroom trials, but I never got. .any 
information." He concluded with: 

"Publishers usually claim materials 
have; been 'classroom tested' or 'u. 
with thousands of students throughout cne 
country ' , but no one should call what they 
do 'testing ' . " 

(This company has no information on 
field-testing available to schools) 



(Company C, Senior Vice President, Editorial) 

"We have about one hundred and sixty salesmen 
and consultants who report back what they pick 
up in the field; that's really our field testing." 



[He then described some testing of the company's 
reading program] . . . "It involved only about twenty- 
fi^^e or thirty kids, but you know if you can find 
that eight out of ten kids don't like a Cc'rtain 
selection, you really don’t have to go further; 

I know this isn't how test people work but it's 
unlikely that you're going to be too far off..." 



"In high school American History, I think we got 
feedback from fifteen schools ... [but] .. .teachers 
don't ask us how we know the books will work... 

I don't foresee any trend toward more field testing 
or evaluation. The 1970 's will see, I feel, a 
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contest between the ’Silberman Camp’ and 
’performance contracts’ . . . [Albert] Shanker 
made a good comment in the Times recently 
and I’ll go along with his position. I 
don’t understand what’s going on in Wash- 
ington these days... And I sure as hell 
would like to know what the hell the 
National Institute for Education is up to.” 

(This company has some field- 
test results available) 



(Company D,Vice President and Editor-in-Chief ) 

’’...mostly we depend on what we hear from 
people out in the field. . .Sure, field-testing 
is good, but it can be overdone. . .Some of the 
government -funded projects are needlessly 
complicated .. .They do their own material and 
revise it and revise it... Any good editor can 
do the same thing, just on the basis of his 
own experience. . .Holt’s Biology still sells 
(this company is not Holt ) after all the 
money they [the governnuent] spent on the BSCS 

materials. . .We didn’t test or 

and they’re still among the best sellers we 
have . ’’ 

(This company’s field-test information 
is for its own use only. We were told 
that if a school wants information on 
field testing and the adoption is im- 
portant enough, an appropriate editor 
will write a letter.) 



In addition to this telephone survey, EPIE also 
conducted an analysis of advertisments for instruc- 
tional materials that appeared in 13 issues of seven 
major educational journals and magazines in recent 
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mo ntrhs . In all, EPIE analyzed 344 advertisements. 

Only 17 contained references to any type of field- 
testing of the product being advertised. An EPIE 
researcher got in touch with the producers who had 
placed the advertisements. Only two of them 
referred our researcher to published research studies; 
six others said they would be willing to supply 
information which ranged from informal feedback from 
class trials to surveys of teacher comments. The 
producers responsible for the nine remaining adver- 
tisements were unable to refer us to (or send to us) 
any evidence to back up the statements made in their 
advertising . 



In one case, it was quite evident that the producer 
had no data of any sort, even though his advertisement 
urged schools to get in touch with any regional office 
"to learn how well these materials are working in 
schools like yours" (This is not the exact 

wording of the advertisement*) further 
investigation ascertained that when a school did, 
in fact, get in touch with a regional office it 
would first be sent a list of schools in that region 
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which had purchased the materials and then it 
would receive a visit from a salesman. 

This sort of sales strategy is, of course, 
disturbing. However, in a sense it was even 
more disturbing to learn that during the months 
that this advertisement was being intensively run, the 
company received only five requests for their 
"field test" information from schools across 
the country! Sales, nonetheless, were quite 
satisfactory during the same period. 



POINT 5.1 



The way schools 
select materials 
does not encouage 
producers to create 
learner- verified 
materials 



When one examines the ways in which most schools 
select materials this lack of attention to evidence 
of effectiveness resulting from learner-verification 
or field testing is not surprising. 



EPIE learned this in 1969 , when it cooperated 
with eight state education departments in surveying 
the evaluation practices used by schools and state 
agencies in the selection of instructional materials 
and equipment. The project identified and studied 
materials selection practices in 19 school 
systems designated by specialists across the country 
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as being more conscien1:ious than many others 
in this task. In every case, these 19 schools 
relied almost completely on examination and review 
of the materials plus (in some cases) discussions 
with sales representatives. Only occasionally 
did selection committees use the results of student 
performance data from pilot tests of the materials 
conducted in local classrooms. Indeed, one state 
department of education included in the study strongly 
recommends local pilot testing of products, but we 
found little evidence that school systems in the state 
follow the suggestion. 



A recent follow-up study of the 19 school systems 
indicates that the practices identified in 1969 are 
still in use today. However, we were somewhat en- 
couraged by the fact that one of these systems does 
press producers for evidence of the learning effect- 
iveness of their products, and that others sometimes 
seek such evidence. As yet none makes such evidence 
a purchasing specification. 



-29- 



I 



POINT 5,2 



Most schools fail 
to use learner- 
verification data 
when selecting 
materials 



When one remembers that these 19 systems were 
designated as being more sophisticated than most others 
in the country in product selection, the nationwide 
picture remains pretty bleak. Nationally, we 
must still conclude that most schools fail to 
employ verification data from learners when 
selecting curriculum materials. 



The reason for this, it would seem, is that in 
evaluating products school selection committees 
must devote most of their limited time to judging 
a product's content and pedagogical approach. 
Practically no time can be given to gathering evidence 
of a material's learning-effectiveness. Committees 
assume that materials with "good content and the 
right approach" will, by definition, be effective 
with learners. Logical as this may sound, it is 
not necessarily true. 



POINT 5.3 



Research supports 
use of learner- 
verification data 
as essential in 
product selection 



Fortunately, some research exists which has ^ 
examined the question of whether it is possible 
to infer the learning effectiveness of particular 
examples of instructional materials by simply 
examining them. This research raises serious 
doubts about the reliability of the practice of^ 
judging the quality of learning effectiveness with- 
out the help of learner-verification data. 




In the first of such studies a group of teachers 

and a principal were asked to review and rank, for I 
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effectiveness^ alternate versions of a set of 
materials on which evidence of effectiveness 
with learners had been gathered by the researchers j 
but was not made available to the educators . With 
no evidence of effectiveness available to them, 
the educators were strikingly unsuccessful in 
judging the learning effectiveness of the materials 
they had attempted to rank. The correlation between 
their judgements and the actual performance of 
the materials with learners was -.75. 



This study is one of few in the literature of 
educational research that has 

had its results corroborated 3. second researcher 
with a similar group of subjects. Despite this 
fact, most school men and members of the education 
industry continue to put their faith solely in 
examination and review rather than evidence of 
actual performance when judging educational materials. 



POINT 5.4 Not just existing research, but common sense, makes 

it quite clear that at present any responsible effort 
The real problem to create or select materials of proven learning effect- 

in schools and iveness must use data from learner-verification. So 

industry the problem is not that research does not answer the 

question directly and unambiguously — the real problem 
, is that the question of learning effectiveness does 

not seem to be of great interest to the producers and 
purchasers of educational materials . This situation 
IS not the result of collusion or conscious negligence 
on the part of companies and schools, but rather 
' of habit, apathy, and ignorance. 
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The fact of the matter is this: many producers 
and purchasers feel they know how to judge the 
learning effectiveness of materials. Thus, they 
become defensive when researchers suggest that 
their methods are less than reliable. Others, 
as we have seen, simply infer that materials 
examined and judged acceptable as to content 
and approach will also produce effective learning. 



The problem, then, is one of changing well-established, 
comfortable behaviors on the part of both producers 
and purchasers. When such is the problem, it is 
difficult to change people by pointing to research 
studies. What, then, can be done to change present 
practices of product development and selection? 



The first step is both simple and difficult. It is 
to admit honestly and candidly that these practices 
can and must be improved. What must be avoided at 
all costs is, on the one hand, preaching a counsel 
of perfection: "Research can^t tell us precisely 
what to do, so let's not change things until it can" 
and, on the other hand, becoming defensive about 
established practices ; "These practices have been 
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POINT 5 . 5 

A new credo 
for professionals 
in schools and in 
companies 



o 




V 



developed and refined through professional 
experience over the years. 



The credo of all professionals -- in the education 
companies and in scnoois — must be: "There isn't a 
product we produce, nor a product now in use, that 
cannot be improved. Every product must continuously 
be revised in light of the growing knowledge and the 
constantly changing needs of learners." 

For example, most producers of materials for young 
children will eventually be forced to revise their 
products for youngsters who have been habitues of Sesame 
Street . This sort of shift in the learner’s general 
knowledge base, brought about in large part by television, 
must continually be accounted for by producers of 
educational materials. 



What is needed is a predo of improvement and 
accomplishment . In the few instances where this 
credo has been followed by action, the results 
indicate that some real progress has been made. 

Let me cite two examples of the sort of improvement 
and accomplishment that can be made when a product 
developer and product user admit that thing can be 
done differently. 

TWO EXAMPLES OF VERIFIED MATERIAL 
The first exeunple I have chosen is a thoroughly 
researched product of a federally-sponsored regional 
educational laboratory. The second is a product 
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from the commercial textbook field that has been 
put through a reasonable approximation of learner- 
tryout over the last two decades. 



The new product is the First Year Communications 
Skills Program developed by the Southwest Regional 
Laboratory for Educational Research and Development. 
The program has 10 units designed for use with 
kindergarten children. The units cover basic 
communications skills (e.g., naming the letters of 
the alphabet, reading simple selections). The 
program takes from 21 to 3 5 weeks for children to 

use, depending on the amount of time allocated daily. 
The producers state: "The reading skills to be 

acquired are listed very explicitly throughout 
the program and their successful mastery is the 
most important result of the instruction". 



This language is similar in tone and content to 
the language one finds in the commercially-developed 
materials but the kind of evidence Oj.Jered by the 
Southwest Laboratory in support of this statement 
is hardly, if ever, forthcoming from commercial 
developers • 
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The heart of the verification procedures used with 
these materials is a special kit that has been 
developed for "Quality Assurance". The contents 
in this kit are related to: (1) mid-year and end- 
of-the-year pupil performance, (2) information 
collection and reporting, (3) school-wide sampling 
plan and schedule, (4) data processing, (5) decision 
rules for selecting alternative courses of action, 
and (6) guidelines for evaluating program modifications . 

Back-up data have been gathered over a four-year 

cycle of product development during which the units ' 

were continuously tried out with learners and 

accordingly revised. The developers readily admit 

that criterion mastery was not achieved when the 

product was first used with children. But now, 

after four years of continuous evaluation and \ 

\ 

revisicn, this mastery has been verified through 

use with more than 30,000 children in 12 states in j 

i 

middle sized schools and several large urban districts. = 
These materials will eventually be commercially 

I 

distributed. 
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The inclusion in the Southern Regional Laboratory's 
kit of a set of guidelines for evaluating program 
modifications prompts me to comment on a study 
which should be of particular interest to this 
Committee. It was recently brought to my attention 
by an educational researcher who discovered it in the 
process of preparing a review of research on the 
nature of product development. The study was 
conducted by the Department of Defense under the 
title of Project Hindsight. 



The relevant finding of Project Hindsight was that 
when researchers conducted a retrospective exam- 
ination and analysis of the functioning of a tech- 
nological system they were frequently able to 
identify a critical part or function that could 
be modified at very little cost. When the identified 
function was, in fact, modified as a result of this 
"hindsight" it was discovered that this low order 
modification of the single variable frequently 
resulted in very high order improvements in the 
overall effectiveness of the system. 



component or 
function 

In my opinion, it is not unlikely that the improvement 
of a critically important part of an educational material 
will bring about an improvement in the overall performanc 
of the material comparable to the improvements achieved 
by Project Hindsight with technological systems. 



POINT 6.1 

D. D. study 
indicates that 
cost-effective- 
ness of technical 
systems are 
frequently im- 
proved thru a 
minor modifi- 
cation in one 
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My second example , a traditional enough looking 
textbook, was first published more than 20 years 
ago yet it continues to outsell all other texts 
in its field. It is an introduction to economics j 
created by Professor Paul Samuelson of M.I.T. 

as a freshman college text, which is also used 
in a few senior high school classes. 

Since the first edition was published in 1948 , 
ceven subsequent editions have appeared and an 
eighth is scheduled. Each edition has gone through 
a three or four-year revision cycle in order to 
update content, organization and style of pre- 
sentation — geared to increasing the effectiveness 
as well as the sales of the material. 

To prepare for each revision, the developer gathers 
information from both teachers and students re- 
garding the teaching-learning effectiveness and 
acceptability of the text. The author has informed 
EPIE that a sampling of institutions using the 
text is selected and questionnaires are distributed 
to students who have used it. 

The questionnaires are returned directly to the 
author for analysis and use in revising the text. 

In addition to this tri-annual feedback from students 
the author continuously monitors the use of the 
materials with a small group of lower-than-average 
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students taught by another professor at a nearby 
university. 



If the Committee wonders why I have selected this 
college text as my second example rather than one of 
the 14,000 elementary and secondary texts now on 
the market, it is because in four years of research 
EPIE has failed to identify one of those 

14,000 texts which has been continuously and system- 
atically revised over the years in a fashion com- 
parable to that used by Professor Samuelson in 
revising his economics textbook. 



POINT 6.2 

All materials should 
be regularly revised 
using learner-ver- 
ification data 
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From the standpoint of the relative size of the 
college and elementary-high school markets there 
is no reason why elementary and high school text- 
books and other materials as well cannot be revised in 
this manner. There is also no reason why learner 
feedback at these lower educational levels could 
not involve actual measures of learning effect- 
iveness obtained by testing students, whereas, 
at the college level, the use of student questionnaires 
requesting information about what they learned and 
failed to learn from the text is probably a justifiable 
alternative to such testing. 
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POINT 6.3 

Effectiveness 
often sacrificed 
for new effects 



Today, there is a growing tendency within the 
education industry and the schools alike to play down 
textbooks in favor of more complex multimedia 
instructional systems. This movement is, in part, 
motivated by the belief that various effects which 
can be achieved by . Im, television, and sound and video _ 
tapes are educationally desirable (It is also true 

that media systems and kits often command higher prices 
and larger profit margins) 



The result is that schools often end up s pend ing more 
money for a multiplicity of effects they may not 
need, but can easily buy, than for learner^ 
effectiveness they cannot afford to do withou t. 
Research to date indicates that no one medium is 
always more effective than any other. This 
evidence has helped to produce the current trend 
toward using a multiplicity of media. However 
such multi-media systems may prove, in many cases 
to be a totally unnecessary brandishing of technology. 



Buckminster Fuller characterizes the achievement of 
technology as "doing more with less." I suggest 
that — given the phenomenal growth and uncriti<^al 
acceptance of all types of educational materials».- 
it just may be t>at we are "doing less with 
with more! " The kind of research and development 
being planned for the National Institute of Ed- 
ucation could help us find out just what we are doing. 
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Without such systematic efforts we will 
continue to fly blind. 



One example of such blind flying is in order. 
Recently a new multimedia course in elementary 
economics was developed for the U.S. Naval Academy 
by a private contractor . The course materials 
included programmed books, nonprogrammed printed 
materials, films, computer simulations and manuals, 
audio cassettes, and student response devices. 
Materials for each student cost approximately 
$100, not including the amortized cost of the films 
and computer simulations (The eleven simulation 
manuals cost an average of $3.50 each) Despite 
the availability of all these media, the majority 
of students opted for learning from the programmed 
bocks alone. Almost half said the films were boring 
and a waste of time. The programmed materials were 
learner-verified; the films were not. 



The Naval Academy investment in this single course 
has been substantial ($643,020) and the continuing 
per student cost is by no means small. The Naval 



o 

ERIC 



40 



Academy might well ask two questions: is the 
multiplicity of media really necessary or simply 
an example of technological overkill? How well 
do these course materials, costing approximately 
$100 per student, compare with Professor Samuelson’s 
textbook costing a small fraction of that amount? 



One thing is certain: We cannot afford the luxury 

of being encouraged by the two examples I have cited — 
or by the few others I could have used; We must 
not forget that an estimated 9 9 percent of the 
materials now being sold to schools have not been 
developed in this fashion. The fact to remember is 
that the 200,000”plus educational materials represented 
by that 99 percent will not suddenly disappear from 
schools, nor will they be taken off the market by 
their producers. Many of these materials are far 
from worthless, but all of them are far from being 
as good as they could be — indeed, must be -- if 
they are to meet education's new needs. If, for 
instance, they were designed with no "target learners" 
in mind then there is an urgent need to discover with 
which learners they are most effective — and on which 
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POINT 6.4 

Yesterday’s tools 
are not good enough 
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learners they are lost. These are persistent* 
unanswered questions that must be dealt with* 
if the ultimated educat-onal consumer — the 
learner^ is to be served. Schools must have 
materials that meet the needs of all our 
children — and that meet them as individually 
as possible. 



As they now stand, the materials schools use 

are not good enough to meet this need, nor ^ ^ 

are they good enough to expect our teachers willingly to 

to be held accountable for what students fail to 

learn when required to use those materials. Schools 

give students and teachers yesterday’s tools and 

expect them to prepare our society for tomorrow. 

Xf the i-esponse of both is increasing unrest, 
should we be surprised? The time to retool education 
is now — before a revived economy and a less 
cautious Congress make educational money once again 
easy to come by . 



Unless action is taken now, schools will inevitably 
spend the money they receive for what is available, 
in other words for the unimproved materials they. are 
now using. What specifically should be done? 
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RECOMMENDATIONS 



POINT 7.1 

Need for 
National 
Institute 



POINT 7.2 

Need for realistic 
guidelines for 
producers and 
schools 



1 
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First, a National Institute of Education is needed 
to institutionalize a continuing broad-based research 
program into the many problems surrounding the develop- 
ment, evaluation, selection and use of educational 
products. This program should place great emphasis 
on the need to improve materials-via learner-verificatior 
It also must try to discover other means of improving 
the effectiveness of educational materials . 



Had a National Institute of Education with such a 

t 

charge been established as part of the National 
Defense Education Act in 1958 or as part of the 
Elementary and Secondary Education Act in 1964, 
there would be no need for me to be here . But 
time, money and learning are being wasted. We must 
improve the tools which 50 million learners and 
two million teachers are required to use. 



The first task of the National Institute of Education 
in such a program should be the development and 
dissemination of realistic guidelines to help 
product developers institute a program of verification 
and revision. These guidelines would be aimed at both 
commercial and non-commercial product developers . 



A second set of guidelines should also be developed 
to assist the schools in selecting materials. Here 
too, great emphasis would be put on euiding schools 
to the purchase of learner-verification materials only 
These guidelines would not eliminate the need for 
independent product evaluations , but they could help 
create a set of minimum standards to be used by in- 
dependent product evaluators . 
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I repeat that these guidelines must be realistic; 

I mean that they must contain realistic recommendations 

and make realistic demands . The guidelines put forth 
for the development and selection of programmed 
instruction materials in 1963 by a joint committee 
of three well-meaning professional associations 
were not realistic. Those guidelines mistakenly ^ 

f 

counseled perfection and advocated a single route 
to that perfection. Needless to sayi the guidelines 
had little effect. 

Educational producers, educational practitioners, 
and for that matter, educational researchers are not 
perfect— but they all can improve. Thus, the guide- 
lines envisioned here are guidelines that would be 
implement able by any producer or any school wishing 
to improve present practice. And they would be 
realistic enough to offer a number of routes to 
achieving improvement, i.e., modest products could 
be evaluated modestly, and more complex and more 
expensive products more ambitiously. Products 
such as total reading programs or entire K-12 curriculum 
programs would receive thorough, continuous learner 
verification on a even larger scale , with great at- 
tention to the task of adcitpting these major programs 
to the changing needs of learners. 
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The specific mechanism for formulating these 
guidelines would be an NIE Technical Task Force 
made up of NIE staff and representative groups 
of specialists in product verification. Such 
specialists employed in the education industry 
should be invited to participate as individuals. 



The working assumption of this Task Force should be that 
all educational materials (with the exception of 
those not designed for the teaching and learning 
of specific instructional objective, i.e., fiction, 
biographies, general background or enrichment materials 
in book or nonbook media) should be continuously 
revised using data from learner- verification. 



The guidelines for schools should help purchasers 
make maximum and efficient use of verification evidence, 
and should urge schools to refuse to purchase 

non-verified materials . 




If this NIE Task Force cannot be formed under 
the aegis of NIE this year, then an interim 
task force should be formed which could work 
under NIE when it beccrmes established. EPIE 



would be happy to serve as a vehicle for this 



task force. 
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Once these guidelines have been developed and 
disseminated, producers would be expected to comply 
with them within a specified period of time. At 
the end of that period, each producer would be 
expected to publish a statement of learner-verification 
evidence for each of his products. 



Obviously, it is going to cost producers money as 
well as time to comply with the proposed guidelines. 
Producers will have a new item to add to their 
product development budgets: the cost of gathering 
and using feedback from learners. Sad to say, this 
will be a totally new experience for most producers. 
This increased cost must inevitably increase the 
costs of materials to schools. But continuously 
improved learner-verified materials must in the 
long run reduce many important non-dollar costs 
that are now being passed on to the learner. 



POINT 7.3 



Federal aid 
for the support 
of learner- 
verified materials 



If these increased dollar costs are too great for 
producers and purchasers to absorb, then 
Federal aid Blight be offered. Federal aid to 
producers could take the form of research and 
development grants to be used to improve specific 
materials through verification and revision. Federal 
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aid to schools could be in the form of increased 
Federal funds for school systems that use 
the proposed guidelines and institutp. purchas tng 
policies that clearly demand learner verified 

materials. Tne outcome of such str<=itegi r^^llv— 
managed Federal funding would be to drive out 
stagnant) unimproved materials and provide schools 
with useful and effective alternatives to what they 
are now using. 



But until such a system is instituted, schools 
should do two things: press producers to supply 

evidence of the learning effectiveness of their 
materials , and indicate their willingness to serve 

as sites for leeU?ner-V6rifica"tion studies. 



POINT 7.4 



Education companies 
and school boards 
share the responsi- 
bility for continued 
use of unimproved 
matf.rials 



Until such time as these recommendations, or a 
reasonable facsimile of them are adopted, educational 
companies that fail to conduct evaluations during 
the development and revision of a product and 
schTOl boards that fail to demand proof of such eval- 
uations share the responsibility for putting unproven 
learning materials into the hands of teachers and 
students . 



I repeat my contention that this situation is 
the result more of habit, apathy , and ignorance than 
of collusion or negligence on the part of companies 
and schools, but I also state that now is the time 
to change the habits that have created the present 
situation. All parties, the industry, the schools, 
amd the Congress must admit to having been ignorant; 
now they must do what must be done to be come wiser . 

The proposed National Institute could do mush to create 
the knowledge upon which such wisdom depends. Millions 
of children stand to learn better if we will allow 
ourselves to create that knowledge and seek that wisdom* 
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I realize I have said a great deal in this testimony 
about the need for learnei -verification and revision 
as a means of improving educational materials, and 
as a means of protecting the learner as the ultimate 
educational consumer. In closing, I want to make it 
clear that I have placed this emphasis on learner- 
verification not because I see it as the only 
means of improving the effectiveness of educational 
materials, but because at present it is a thoroughly 
effective me>ii>s of improvement at the 
disposal of both producers and schools — which is 
largely ignored. 
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